Adapting to the User,s Time and Working Memory Limitations: New Directions of Research

As interactive computing devices become increasingly mobile, the user,s time limitations and situational distractions are becoming important determinants of the quality of interaction with these devices. Automatic adaptation to such situational constraints is a challenging goal for user modeling research. This paper first briefly summarizes research in the first three-year phase of the READY project, in which a prototype of such a system was developed. It then discusses four new lines of research that will be pursued in this project during its second phase: extension to multimodal interaction; allowing for different degrees of granularity in the modeling of the user; learning the system,s Bayesian networks automatically from empirical data; and dealing with more complex discourse structures. Computing power is moving off of the desktop and into the hustle and bustle of everyday life. This trend generates new challenges and opportunities for those who develop adaptive systems. Users of a desktop computer may have to cope with deadlines and distractions, but on the whole they can measure their time in minutes and hours, rather than in seconds; and they can typically concentrate fully on whatever task they are using the computer for at the moment. By contrast, some of the more recent additions to the family of computing devices— such as hand-held computers and smart mobile phones—often constitute just one element in a complex environment that the user has to deal with. An interaction may be have to be completed within seconds, while the user is simultaneously conducting a conversation and/or navigating through unfamiliar territory. In such a situation, two critical factors that influence the success of interaction with the system are: 1. the amount of time that the user (U ) has available; and 2. the extent to which U can concentrate on the interaction, as opposed to dealing with situational distractions. Somewhat more abstractly, we can speak of bottlenecks caused by temporary limitations of the user,s resources of time and working memory.1 It is not obvious that the user,s temporary resource limitations need to be recognized and adapted to automatically. First of all, systems of this sort are typically designed from the start in such a way that they can be used quickly and without much mental effort. But no single type The research outlined in this paper will be funded from 1999 though 2001 by the German Science Foundation (DFG) in its Collaborative Research Center on Resource-Adaptive Cognitive Processes, SFB 378, Project B2, READY. The principal investigators are Wolfgang Wahlster and Anthony Jameson. The complete text of the research proposal (Wahlster & Jameson, 1998a) is available on request. The project Web page is http://w5.cs.uni-sb.de/ ̃ready/. 1The Collaborative Research Center on Resource-Adaptive Cognitive Processes, where the research discussed here is being conducted, investigates ways in which the performance of cognitive systems adapts to limitations in resources of various types (see http://www.coli.uni-sb.de/sfb378/). of interaction is optimal for all configurations of resource availability. For example, if a system is designed for the worst case of a rushed and distracted user, it will probably make too little use of the user,s resources when more of these are available. What about the strategy of making these systems adaptable rather than adaptive? For example, the designers could give users the opportunity to choose, in each situation, how they would like to enter their input to the system and what type of output they desire. Unfortunately, a user who is rushed and/or distracted is in a poor position to adapt his system use accordingly—or even to signal the problem to the system. In sum, it seems to be a reasonable goal to enable some types of interactive computing devices to do what people do when interacting with others: The device should assess, without being told, roughly what resources the person has available, and it should adapt its behavior accordingly. To be sure, realizing such automatic adaptation is a challenging task, for the following reasons, among others: 1. The evidence that can permit a system (S) to recognize resource limitations tends to be scant and not very reliable. A usefully accurate assessment is often only possible when several pieces of evidence can be combined. 2. The temporal variability of these resource limitations means that S is tracking a poorly visible moving target. 3. Even if the user,s resource availability is perfectly known, it is often hard for a system to decide what type of adaptation is appropriate. For example, an especially concise instruction may initially save a user some time, but if it is too dense do be understood completely, it may in the end cause the user to waste time or make errors. The READY project, which began in 1996, represents an initial effort to meet these challenges. This paper will first briefly summarize the research of the first three-year phase.2 It will then discuss the new directions of research that will be pursued during the second phase. The Current Prototype The example scenario that was used during the first phase of research is illustrated in Figure 1: Users are drivers whose cars need minor repairs; they request assistance from the system in natural language via mobile phone. Exploratory empirical studies in this scenario (and with a different scenario involving emergency calls to a fire station) provided some background information concerning the following points: Causes of resource bottlenecks: factors (such as the difficulty of the current task, or the need to perform another task simulteneously) that create a situation in which the user,s resources may not be adequate for successful interaction. Symptoms of resource bottlenecks: aspects of the user,s behavior (e.g., speech disfluencies) that indirectly indicate the extent to which U ,s resources are adequate to meet the current demands. Strategies for dealing with resource bottlenecks: general ways in which systems (or persons) can adapt their behavior when a resource bottleneck is perceived. 2Jameson, Schäfer, Weis, Berthold, and Weyrath (in press) give a recent English-language overview of the research of this phase. Schäfer, Weis, Weyrath, and Jameson (1997) presented an earlier overview in German. The report by Wahlster and Jameson (1998b) is available on request. Papers presented at previous ABIS workshops include those by Schäfer and Weyrath (1996) and by Berthold (1997). Figure 1: Illustration of READY,s first example scenario, showing the similarity of READY,s role to that of an auto repairman who offers advice by phone. These studies guided the development of the initial READY prototype. The basic workings of the prototype can be explained with reference to the sketch of the architecture in the left-hand side of Figure 2. The User Interface is designed so as to make it unnecessary for READY to deal with the challenging problems of speech processing raised by the scenario. Input is done via a natural language menu interface with which the “user” can compose utterances and specify a number of aspects of their form, such as the position and length of pauses. What is sent to the Dialog Management component is a representation of U ,s utterance that contains the information that READY needs in order to update its user model and determine an appropriate response, i.e.: the meaning of U ,s utterance in the current dialog context plus any evidence that can be extracted from the input (including noises from the environment) that bears on U ,s current resource limitations. The Dialog Management component uses its knowledge about the current dialog state and about possible diagnosis and repair actions to determine a set of possible dialog contributions (e.g., instructions) that might make sense in the current situation—not yet taking into account U ,s resource limitations. These contributions may differ in their basic content (e.g., prescribing a simple or a complex action) and/or in their form (e.g., using simple, redundant formulations or concise, technical ones). The Dialog Management component then sends the possible contributions to the User Modeling component, which decides which one seems best in the light of U ,s current resource limitations. The User Modeling component employs dynamic Bayesian networks as a representation and processing formalism, because they are well suited to dealing with (a) the large amount of uncertainty involved in inferences about a user,s resource limitations and (b) the dynamic nature of these resource limitations, which often change quite rapidly. Network Schemata represent assumptions about the causal relationships between resource limitations and their causes and symptoms. Figure 3 gives an idea of the way in which this prototype adapts to a “user,s” perceived resource limitations. The numbers on the right are the expected values of the system,s current assessments of key variables, including available WM capacity (vAGK) and time pressure (Zeitdruck). In this case, the user (P) has just input the problem description Kühlwasser-Warnlampe